Per-project test execution, and fork-death diagnostics that name a cause - #686
Open
oyvindberg wants to merge 18 commits into
Open
Per-project test execution, and fork-death diagnostics that name a cause#686oyvindberg wants to merge 18 commits into
oyvindberg wants to merge 18 commits into
Conversation
Make per-project (maven's forkCount=1 reuseForks=true) the default test JVM mode, replacing per-suite. An unset testJvm now runs a project's suites in one shared fork; testSuiteParallelism bounds concurrency inside it and defaults to ~cores/4 (per-suite still means a fork per suite, unbounded by default). The machine-wide governor caps the total across projects either way. The default is resolved in one place (testJvm.getOrElse(PerProject)) at the two sites that read it, and suiteParallelism's default is now mode-aware so both the JUnit-Platform batch path and the shared suite-by-suite path get ~cores/4 in per-project mode. Docs-driven: adds docs/usage/testing.mdx (the execution model, when to reach for each mode, JVM options, selecting what runs), registers it and the existing test-tags page in the sidebar, adds the missing testJvm field to schema.json, and refreshes the model scaladoc and testSuiteParallelism's stale description. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtKgAYQtNSjkyDWVb4UyTr
Genericize every Quarkus reference in the test-execution code, comments, docs, and test data — the general "run a project's suites in one JVM" feature stands on its own, described in terms of the mechanisms it uses (a singleton-per-JVM application, a LauncherSessionListener, shutdown hooks that stop containers) rather than one framework. The Quarkus-specific naming and rationale belong in the later Quarkus PR, which is where `@QuarkusTest` actually enters the build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtKgAYQtNSjkyDWVb4UyTr
…form The per-project shared-session support calls JUnitPlatformRunner.enableSharedSession() (and closeSharedSession() in the finally) unconditionally at fork startup. Both merely reference JUnitPlatformRunner, which links against org.junit.platform.launcher.* — and junit-platform-launcher is a `provided`, compile-only dependency of bleep-test-runner, so it is NOT on the runtime classpath of a fork for an sbt test-interface project (ScalaTest, MUnit, utest, Specs2, ScalaCheck, weaver, ZIO Test). Referencing the class there throws NoClassDefFoundError before the fork sends Ready, so every one of those suites failed with "Expected Ready, got Exception in thread main" — i.e. the whole feature broke the majority of Scala test projects. Guard both calls with a lazy Class.forName probe for a launcher class: touch JUnitPlatformRunner only when JUnit Platform is actually present. An sbt-interface fork has no session to share and none to close, so skipping is correct, not a fallback. JUnit-Platform forks are unaffected. Also genericizes the two remaining Quarkus mentions in this file (they belong in the later Quarkus PR). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtKgAYQtNSjkyDWVb4UyTr
… fork Per-project mode now runs a project's suites of one framework through a single execution — the sbt interface's one-runner-per-framework contract, maven's forkCount=1 reuseForks=true. Before, a shared fork created a fresh Framework/Runner (and called done()) per suite, which corrupted stateful frameworks (munit, ZIO, hedgehog), and the JUnit "batch" ran one launcher.execute() over all classes, which lost per-class attribution for engines whose test tree is not keyed by the selected class (cucumber features, spek specs reported zero). - SuiteRunner: runOneSuiteOn(framework, runner, ...) + runSuites(...) — one Runner for all a framework's suites, done() once, sequential by default with a degree knob. - JUnitPlatformRunner.runSuites: one launcher.execute PER class on the shared LauncherSession (session-scoped fixtures — a @QuarkusTest app — still build once), restoring per-class attribution. - Protocol RunSuites generalized to carry frameworkClass + args and accept sbt-test-interface; ForkedTestRunner routes JUnit vs sbt. - DAG groups a project's JVM suites by framework into one batch each (JUnit and each sbt framework); PlatformRunner (JS/Native) suites stay per-suite. Default degree 1 (sequential — the safe maven default); testSuiteParallelism opts into concurrency. - Output attribution: a fork-level active-suite fallback tags output from a framework's own threads (ZIO fibers, specs2 workers) when the thread-local is unset, so a batched suite's <system-out> is not lost. Matrix (JvmScala213TestFrameworkIT) goes 6/18 -> 16/18 under per-project; weaver and hedgehog still drop a failing test's late output (follow-up). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtKgAYQtNSjkyDWVb4UyTr
…ehog) A few sbt frameworks report a suite's failure detail from Runner.done() — called once per run — or on their own threads after the suite's tasks return. Sharing one Runner across a project's suites (one done(), per-project mode) drops that per-suite output: weaver's <system-err> and hedgehog's <system-out> lost the failing test's reason. Give such a framework its own fork per suite (own done()) regardless of the project's mode, via a small class-keyed denylist (FrameworkSelection.needsIsolatedFork); they are left out of batching and their suite tasks force an exclusive fork. JvmScala213TestFrameworkIT is now 18/18 under per-project (every framework). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtKgAYQtNSjkyDWVb4UyTr
Correct the testing guide, scaladoc, and schema now that per-project's default degree is 1 — suites run one at a time in the shared fork, exactly maven surefire's reuseForks=true, the safe default since not every framework's engine is re-entrant. testSuiteParallelism > 1 opts into concurrency. Also fix the guide's now-stale description of the JUnit path: it runs one launcher execution per class on a shared LauncherSession (session-scoped fixtures build once, per-class attribution exact), not a single execution over all classes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FtKgAYQtNSjkyDWVb4UyTr
…Fork/maxConcurrentSuites The safe default: in per-project mode, sbt-interface frameworks always run their suites sequentially through one Runner/one done(). They share a Runner and have no lock-aware scheduler, so concurrency in a shared JVM is unsafe (shared statics, unattributable async output). maxConcurrentSuites raises concurrency only for JUnit Platform, whose engine owns a lock-aware scheduler; concurrency for sbt suites is testFork: per-suite (a fork each, OS-isolated). Per-project default concurrency is now 1 (was ~cores/4) — memory-frugal, one live suite's heap at a time; cores are saturated across projects by the governor. An sbt-only project that sets maxConcurrentSuites > 1 gets a BSP build/logMessage warning that it has no effect. Renames (public YAML surface): testJvm -> testFork (values unchanged), testSuiteParallelism -> maxConcurrentSuites; type TestJvmMode -> TestForkMode. Cleanup: removed the now-dead concurrent (thread-pool) branch from SuiteRunner.runSuites — sbt batches always run at degree 1 — and its degree parameter. Also fixed two pre-existing stale DiscoveryResult(batchParallelism=) references that no longer compiled (the field is `batches`). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ForkedTestRunner's command loop wrapped `in.readLine()` in the same catch as command dispatch and, on any exception, looped straight back to reading. When the parent (bleep) goes away — it gave up on the fork, or the client disconnected — the protocol socket breaks and readLine() throws on every call, so the fork spun at ~100% CPU forever, re-encoding the same error each iteration. Observed in the wild: a fork stuck ~2h with `main` pegged in TestProtocol.encodeError. These zombies accumulate, starve CPU/RAM, and then freshly-spawned forks get killed at startup — a self-inflicted cascade that looks like "N suites never reported a result" with no cause. Read the socket in its own try: a read failure (or EOF) means the parent is unreachable, so exit and let the finally block clean up. Per-command dispatch errors are still reported and tolerated, now bounded by a consecutive-error cap so no other persistently-failing state can spin either. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…it unit-testable When a test fork never connects back, bleep already tried to quote what it wrote — but read only the bytes available() at the instant the process was seen dead, which is almost always zero: the JVM's "Unrecognized VM option ...", "Could not create the Java Virtual Machine", or an hs_err pointer is flushed a beat later. So the reason was dropped and the failure surfaced as a bare "N suites never reported a result" — the common case (a bad JVM option, a startup crash) hidden behind the mechanism meant for the uncommon weird one. Split reading by whether the fork has exited: an exited fork's writer is closed, so drain its streams to EOF (cannot block) and get the whole tail; a still-alive fork is read non-blocking as before. Moved describeChildOutput / drainAvailable / drainToEof to object JvmPool as private[testing] pure functions taking a bare Process, so DescribeChildOutputTest exercises the real capture (java with a bad option → its stderr) without standing up a pool or the BSP server. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a batched fork stopped after reporting some suites but not others, the result was a bare "batched suites produced no result: <names>" — cause unknown, a dead end. But a fork that dies mid-run of an OutOfMemoryError, a native crash, or System.exit records the reason on its OWN stderr (fd 2), which the per-suite protocol never carries. Drain that stderr into the failure, note whether the fork is still alive (wedged) vs exited, and add a thread dump when it is wedged. "N never reported a result" becomes a lead instead of a wall. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…exit A test that calls System.exit()/Runtime.halt() ends the whole forked JVM; on JDK 24+ (no SecurityManager) bleep cannot block it, and every suite that had not reported is lost. The parent saw only a clean exit-0 and could do no better than "likely System.exit()". Register a shutdown hook that fires only on such an unrequested shutdown (the command loop had not exited on its own terms) and writes a full thread dump to fd 2 — the thread parked in java.lang.Shutdown / Runtime.exit is the caller. The parent already drains fd 2, and the batch paths now append that (forkDeathDiagnostic) to the failure, so "N never reported" finally names the culprit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…gnostic, and every bleep-initiated kill is logged A fork that exits from under a run — a test's System.exit, or a Runtime.halt that skips shutdown hooks entirely — left the parent with a bare "exited 0" and no cause, because whatever it wrote to stderr races the pipe teardown and is routinely lost. - The fork now writes its exit diagnostic (whether the command loop had ended on its own terms, which suites were still running, a full thread dump) to a FILE the parent passes in and reads after death. A file survives the teardown that loses stderr, and its ABSENCE is itself a signal: no file means no shutdown hook ran — a Runtime.halt or hard kill, not a System.exit. - The command loop records WHY it ended (EOF / IOException / Shutdown / error-cap) and, on teardown, names the suites still running that it is abandoning. - JvmPool.kill — the single socket-close chokepoint — logs every bleep-initiated kill with pid, liveness and reason, so a fork's death that is NOT preceded by such a line is known to be self-inflicted rather than bleep's doing. - describeExit no longer asserts "likely System.exit()" for a clean exit-0; it names the three possibilities and points at the exit log to disambiguate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ZXWBMqWCxQzB2yCKLDZ7j
…the client Reflecting over a project's compiled classes (getDeclaredMethods / getAnnotations / Class.isAssignableFrom, in both the fingerprint and annotation strategies) can land on a class whose supertypes or annotations name a type no longer on the classpath — almost always an orphaned .class an incremental compile left behind after a package rename. That throws NoClassDefFoundError, a LinkageError, which is an Error, not an Exception — so scala.util.Try / NonFatal do NOT hold it. It escaped discovery, tore down the whole client handler, and surfaced to the user as "BSP server connection lost (server may have crashed)", pointing at memory when the cause was a stale build. One un-reflectable class now costs only that class: matchFingerprint and detectFrameworkByAnnotation run under a guard that catches LinkageError (separately from NonFatal, so OutOfMemoryError still propagates), logs which class was skipped and that `bleep clean` on that project is the likely fix, and lets discovery continue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ZXWBMqWCxQzB2yCKLDZ7j
…per channels, not raw System.err The fork-lifecycle and failure diagnostics added earlier printed straight to System.err, which lands in the daemon log but without the level/context framing the rest of the server logs with. Route each to where it belongs: - Fork kills (the JvmPool.kill chokepoint) become a JvmPoolListener.onForkKill event, recorded by BspMetrics as a `fork_kill` entry — joined to fork_end/fork_start by pid, so "did bleep end this fork, or did it exit itself?" is answerable from the metrics stream (a fork_end with no preceding fork_kill was not bleep's doing) rather than by grepping raw stderr. Keeps bleep-core logger-free: the pool announces via the listener it already has, exactly as onForkStart/onForkEnd do. - The batch fork-error / fork-diagnostic messages (TestRunner) and the stale-class discovery skip (ClasspathTestDiscovery) now go through ryddig's Logger — threaded in as a parameter — so they carry a level and read like every other server log line. Behaviour is unchanged; only the destination and framing. The fork-side diagnostics (fd 2 + the exit-log file) stay as they are — a separate process has no other channel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ZXWBMqWCxQzB2yCKLDZ7j
The test-execution model added fields to the project model (testFork, maxConcurrentSuites) and schema, which flow into the resolved build the snapshot tests cache. Regenerate the caches so the snapshots reflect the current model. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ZXWBMqWCxQzB2yCKLDZ7j
This PR keeps Quarkus specifics out (they enter in the follow-up that wires @QuarkusTest into the build), but the testing guide still used @QuarkusTest as the session-scoped-fixture example — a reader cannot use it yet. Describe the fixture by what it is (a booted application, a LauncherSessionListener), matching how the code and its comments already phrase it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012ZXWBMqWCxQzB2yCKLDZ7j
…perties
bleep sets no JUnit configuration parameters — the discovery request is
`request().selectors(selectClass).build()` — so a junit-platform.properties on the
classpath still governs parallelism WITHIN a class, and bleep neither enables nor
overrides it. maxConcurrentSuites is bleep's own thread pool running that many classes
at once, each as its OWN launcher.execute; a stale docstring claimed the opposite
("configuration parameters set here override any junit-platform.properties").
Also record the consequence the old text implied away: because each class runs in a
separate execution, @ResourceLock is not coordinated ACROSS classes — keep
maxConcurrentSuites at 1 when a project's classes share mutable state. (Within a class,
jupiter's own parallel execution does honor the locks, if you enable it via properties.)
And genericize the last @QuarkusTest mention in this file's comments (Quarkus stays out
of this PR).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ZXWBMqWCxQzB2yCKLDZ7j
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test running grows up: a real execution model, and forks that can't die in silence
This turns
bleep testfrom "spawn a JVM per suite and hope" into a proper, configurable execution model — and, just as importantly, makes a forked test JVM incapable of dying without telling you why. Two halves: a small, well-documented config surface you'll actually reach for, and a big pile of hardening so a weird test never again shows up as a mysterious "server may have crashed."Part 1 — how a project's tests run, in your control
Everything is driven by ordinary project fields. Here's the full surface, in one place:
Frameworks are auto-detected from the classpath — ScalaTest, JUnit 4/5, MUnit, utest, Specs2, ScalaCheck, weaver, ZIO Test, Kotest, Cucumber, jqwik and more. You usually write just
isTestProject: true.testFork— where suites run, and what that costs youper-project(default)per-suiteforkCount=1 reuseForks=true)System.exit, leaks a thread, or corrupts a static takes down only its own process, which bleep replacesmaxConcurrentSuites(JUnit Platform only)maxConcurrentSuitesbounds how many forks run at onceDefault is
per-project+ sequential. That's the memory-frugal, correct-for-everything choice; your cores stay busy because bleep runs many projects' forks at once under a machine-wide governor, not many suites inside one fork.maxConcurrentSuites— how many at once, and the one ruleIn
per-projectmode this only speeds up JUnit Platform suites: the JUnit engine runs that many of the project's classes at once inside the one fork, each as its own launcher execution. bleep sets no JUnit configuration parameters, so ajunit-platform.propertiesyou ship still governs parallelism within a class (@ResourceLock/@Executionand all) — bleep neither enables nor overrides it. Because each class runs in a separate execution,@ResourceLockis not coordinated across classes, so keepmaxConcurrentSuitesat1when a project's classes share mutable state. That's deliberate — JUnit is the one runner whose ecosystem is built for parallel execution.What is NOT allowed / has no effect:
Runner, expose no conflict graph, and generally aren't written for it. SetmaxConcurrentSuites > 1on an sbt-only project and it's a no-op — bleep emits a BSP warning telling you so. To run sbt suites concurrently, switch totestFork: per-suite(a fork each, safely isolated).maxConcurrentSuites; sbt → concurrency viaper-suite. One rule, and everything else follows.Why
per-projecthonors stateful frameworksThe sbt test interface's contract is one
Framework/Runnerper framework,done()called once per run.per-projectruns a project's suites of a framework through exactly that — oneRunner, onedone(). Frameworks that keep run-level state or report a suite's detail fromdone()(weaver's<system-err>, hedgehog's<system-out>) depend on it; give each suite its own freshRunner/done()and that reporting is lost. Frameworks that can't share aRunnerat all are detected and given their own fork per suite automatically (weaver, hedgehog today). Result: the framework matrix goes from 6/18 → 18/18 green under per-project. JUnit Platform gets the analogous treatment — onelauncher.executeper class on a single sharedLauncherSession, so a session-scoped fixture builds once and per-class attribution stays exact.Part 2 — a forked JVM can no longer die in silence ⭐
This is the part that will save you a debugging session. Previously, a test fork that died oddly surfaced as "BSP server connection lost (server may have crashed)" — which points at memory, when the real cause was a stale build, a bad
-Xmx, or a test callingSystem.exit. Every one of those is now caught and named.Abandoned forks exit instead of spinning forever. A fork whose parent went away used to loop on a broken socket at ~100% CPU, re-encoding the same error — one was found pegged for ~2 hours. These zombies pile up and starve the machine, which then kills fresh forks at startup: a self-inflicted cascade that looked like "N suites never reported a result." The command loop now exits on a broken/closed socket, bounded so no failure state can spin.
Death without finalizers is detected and distinguished. A test that calls
Runtime.halt()— or a nativeexit()— kills the JVM running no shutdown hooks at all, so nothing it writes to stderr survives the pipe teardown. The fork now writes its exit diagnostic (did the command loop end on its own terms? which suites were still running? a full thread dump) to a file the parent hands it and reads after death. The file's absence is itself the signal: no file ⇒ no hook ran ⇒ it wasRuntime.halt/a hard kill, notSystem.exit. That single bit ends the guessing.System.exitnames its caller. When a test ends the JVM the ordinary way, a shutdown hook dumps every thread to fd 2 — the one parked injava.lang.Shutdown/Runtime.exitis the culprit — and the parent folds that into the failure.A failed fork's stderr is actually captured. We now drain an exited fork's streams to EOF instead of reading the zero bytes
available()at the instant of death. That's what finally surfacesUnrecognized VM option …,Could not create the Java Virtual Machine, andhs_errpointers — the common startup failures that used to vanish."N suites never reported a result" became a lead, not a wall. A batch fork that stops mid-run now reports the drained stderr, whether the JVM is still alive (wedged) vs. exited, and a thread dump when wedged.
Every bleep-initiated kill is recorded.
JvmPool.kill— the single chokepoint every eviction/timeout/cancellation/shutdown passes through — now emits anonForkKillevent (pid, reason, was-it-alive), recorded on the fork-lifecycle stream and joinable tofork_start/fork_endby pid. So "did bleep end this fork, or did it exit itself?" is answerable after the fact: afork_endwith no precedingfork_killwas not bleep's doing.Test discovery no longer kills the client over a stale
.class. Reflecting over compiled classes can hit an orphaned.classleft by an incremental compile after a package rename, whose supertype no longer exists →NoClassDefFoundError, aLinkageErrorthatTry/NonFataldo not hold. It used to tear down the whole client. Now one un-reflectable class costs only that class: it's skipped, logged (with "bleep cleanon that project clears it"), and discovery continues.Diagnostics go through the right channel. The parent-side breadcrumbs are structured (
Logger) or recorded as metrics events, not rawSystem.err— levels, context, and joinable records instead of loose lines.And a floor under all of it: a broken build can no longer render as a green "0 tests passed", and a kill the user requested is reported as a cancellation, not a failure.